
United States Patent and Trademark Office 



UNITED STATES DEPARTMENT OF COMMERCE 
United States Patent and Trademark Office 

Address: COMMISSIONER FOR PATENTS 
P.O. Box 1450 

Alexandria, Virginia 22313-1450 
www.uspto.gov 



APPLICATION NO. 



FILING DATE 



FIRST NAMED INVENTOR 



ATTORNEY DOCKET NO. 



CONFIRMATION NO. 



09/662,679 



09/15/2000 



20985 7590 11/16/2004 

FISH & RICHARDSON, PC 
12390 EL CAMINO REAL 
SAN DIEGO, CA 92130-2081 



Fernando CM. Martins 



10559/I95001/P8367 



1908 



EXAMINER ' 



BECKER, SHAWN M 



ART UNIT 



PAPER NUMBER 



2173 

DATE MAILED: 11/16/2004 



Please find below and/or attached an Office communication concerning this application or proceeding. 



PTO-90C (Rev. 10/03) 



Office Action Summary 


Application No. 

09/662,679 


Applicant(s) 

MARTINS, FERNANDO C. M. 


Examiner 

Shawn M. Becker 


Art Unit 

2173 





- The MAILING DATE of this communication appears on the cover sheet with the correspondence address •• 
Period for Reply 



A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1.136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1)S Responsive to communication(s) filed on 12 October 2004 . 
2a)D This action is FINAL. 2b)^ This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213. 

Disposition of Claims 

4) K Claim(s) 1-3,7-22,26 and 28-30 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) M Claim(s) 1-3, 7-22,26 and 28-30 is/are rejected. 

7) D Claim(s) is/are objected to. 

8) Q Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) Q The specification is objected to by the Examiner. 

10) D The drawing(s) filed on is/are: a)D accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). 
Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 

1 1) D The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

1 2) D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 1 9(a)-(d) or (f). 
a)D All b)D Some * c)D None of: 

1 .□ Certified copies of the priority documents have been received. 

2. Q Certified copies of the priority documents have been received in Application No. . 

3. D Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 



Attachment(s) 

1) ^ Notice of References Cited (PTO-892) 0 

2) CD Notice of Draftsperson's Patent Drawing Review (PTO-948) 

3) Q Information Disclosure Statement(s) (PTO-1449 or PTO/SB/08) 

Paper No(s)/Mail Date . 



4) CD Interview Summary (PTO-413) 

Paper No(s)/Mail Date. . 

5) (HI Notice of Informal Patent Application (PTO-1 52) 

6) □ Other: . 



U.S. Patent and Trademark Office 

PTOL-326 (Rev. 1-04) 



Office Action Summary 



Part of Paper No./Mail Date 20041 108 



Application/Control Number: 09/662,679 Page 2 

Art Unit: 2173 

DETAILED ACTION 

This action is in response to Request for Continued Examination filed 10/12/04. 

Claim Rejections - 35 USC §103 



1. The following is a quotation of 35 U.S. C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

2. Claims 1-3, 7-22, and 26 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
U.S. Patent No. 6,227,968 to Suzuki et al. (hereinafter Suzuki) and U.S. Patent No. 6,256,033 to 
Nguyen (hereinafter Nguyen). 

Referring to claims 1 and 26, Suzuki teaches a method and computer program product 
that receives audio data including a beat and extracting the beat data from the audio data (i.e. see 
col. 7, lines 24-34, which describes how beat data is correlated with audio data/songs, implying 
that the beat data from each song must be extracted for proper correlation). Suzuki teaches 
determining a gesture window within which a gesture (dance move) should occur, based on a 
specified time window relative to the beat data, and Suzuki plays the audio data and obtains 
gesture data during a time that the audio data is being played. See col. 7, lines 35-50. Also, see 
col. 1, lines 9-13, which describes how the player is to perform the action in time with the 
rhythm (i.e. the method checks if the player made the proper gesture in time with the beat). 

Suzuki segments the gesture data according to the specified time window (time between 
beats), and Suzuki automatically determines whether a predefined gesture occurred within the 
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specified timing window (i.e. if the appropriate dance move/gesture occurred within the timing 
window for score keeping). See col. 7, lines 25-55 and col. 10, line 30 - col. 1 1, line 37. 

Suzuki does not teach that the gesture data is obtained through video data, wherein the 
video data is segmented to create a video clip of time including specified time window. 

However, Nguyen discloses a method and a computer program product for recognizing 
gestures contained in video data that segments video data to create a video clip based on timing 
data that indicates a specified timing window within which a gesture will occur. See col. 2, lines 
23-25, which states, "The frame captures the person in the action of performing the gesture at 
one moment in time" and col. 2, lines 28-34, which states, "These sequence of frame data sets 
taken over a period of time. . ." The method determines information related to a gesture 
occurring in the video clip only at the specified timing. See col. 3, lines 11-14, which describes 
"determining particular coordinates of the subject at a particular time ", (emphasis added). 

It would have been obvious to one of ordinary skill in the art to substitute the gesture 
recognition method (step-on base of a floor panel) in the dance game of Suzuki with the video 
gesture recognition method of Nguyen, such that the video data is segmented into video clips of 
the specified timing of Suzuki according to beat data as supported by the beat timing window of 
Suzuki, because the step-on base of Suzuki only captures the movements of the player's feet, and 
does not recognize the movements of the rest of the body as intended in Suzuki (col. 1, lines 44- 
51), wherein the video gesture recognition method of Nguyen captures the movements of the 
entire body. 

Referring to claims 14, Suzuki teaches a system with an audio part that receives audio 
data including beat data and extracting the beat data from the audio data (col. 7, lines 24-34). 
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Suzuki teaches determining a gesture window within which a gesture (dance move) should 
occur, based on a specified time window relative to the beat data, and Suzuki plays the audio 
data and obtains gesture data during a time that the audio data is being played. See col. 7, lines 
35-50. Also, see col. 1, lines 9-13, which describes how the player is to perform the action in 
time with the rhythm. 

Suzuki segments the gesture data according to the specified time window (time between 
beats), and Suzuki automatically determines information related to a gesture occurring within the 
specified timing window (i.e. if the appropriate dance move/gesture occurred within the timing 
window for score keeping). See col. 7, lines 25-55 and col. 10, line 30 - col. 1 1, line 37. 

Suzuki does not teach that the gesture data is obtained through video data, wherein the 
video data is segmented to create a video clip of time including specified time window. 

However, Nguyen discloses a system with a processor (Fig. 1, 102) for recognizing 
gestures contained in video data that has a temporal segmentor that segments video data to create 
a video clip based on timing data that indicates a specified timing window within which a gesture 
will occur. See col. 2, lines 23-25, which states, "The frame captures the person in the action of 
performing the gesture at one moment in time" and col, 2, lines 28-34, which states, "These 
sequence of frame data sets taken over a period of time. . ." The method determines information 
related to a gesture occurring in the video clip only at the specified timing. See col. 3, lines 11- 
14, which describes "determining particular coordinates of the subject at a particular time ", 
(emphasis added). Nguyen also discloses a recognition engine, in communication with the 
temporal segmentor, to determine if the video clip contains a predefined gesture, only within a 
specified timing window. See col. 1, lines 17-23 and col. 3, lines 11-14. 
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It would have been obvious to one of ordinary skill in the art to substitute the gesture 
recognition system (step-on base of a floor panel) in the dance game of Suzuki with the video 
gesture recognition system of Nguyen, such that the video data is segmented into video clips of 
specified timing according to beat data as supported by the beat timing window of Suzuki, 
because the step-on base of Suzuki only captures the movements of the player's feet, and does 
not recognize the movements of the rest of the body as intended in Suzuki (col. 1, lines 44-51), 
wherein the video gesture recognition system of Nguyen captures the movements of the entire 
body. 

Referring to claims 2-3, the combination of Suzuki and Nguyen, supra, describes how 
Hidden Markov Models are used to determine a probability that each of one or more predefined 
gestures are performed within a timing window. See col. 5, lines 13-45 of Nguyen. 

Referring to claim 7, 17-18, and 21, the dance game of Suzuki teaches displaying a target 
gesture to be performed by the subject. The target gesture is a dance move. See col. 12, lines 
57-61, The target gesture is displayed on the display subsystem (monitor; col. 12, line 59). 

Referring to claims 8-10, the combination of Suzuki and Nguyen, supra, teaches each 
video clip contains video frames (Nguyen at col. 7, line 22), and in each frame, the moving 
regions are identified (Nguyen at col. 8, lines 58-61). Feature vectors (array of key points) are 
generated for each video frame of the video clip. See Nguyen at col. 8, line 49 - col. 9, line 46, 
which describes how significant positional coordinates are extracted from each frame to make a 
comparison to the known gesture coordinates. 

Referring to claims 11-12 and 19-20, the method of Suzuki and Nguyen, supra, generates 
and displays a score based on whether a target movement (gesture) was performed. See Suzuki 
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at col. 1 1, lines 37-41 . The object of the Suzuki dance game is to achieve the highest score by 
performing the appropriate gestures. 

Referring to claims 13 and 22, the combination of Suzuki and Nguyen, supra, discloses 
that the recognition engine is configured to recognize predefined gestures and that determining if 
the video clip contains a target gesture includes generating a gesture probability vector (array) 
having a plurality of elements, each element being associated with one of a predefined gestures 
and representing a probability that the video clip contains each of the associated predefined 
gestures. See col. 11, lines 30-49 and col. 10, lines 18-37. 

Referring to claim 15, the recognition engine of Suzuki and Nguyen, supra, includes a 
plurality of Hidden Markov Models. See Nguyen at col. 5, lines 13-18. 

Referring to claim 16, the system of Suzuki and Nguyen, supra, includes a video source 
(Nguyen at Fig. 2, 200), in communication with the temporal segmentor, to provide the video 
data to the temporal segmentor. 

3. Claims 28-30 are rejected under 35 U.S.C. 103(a) as being unpatentable over Nguyen and 
Suzuki. 

Referring to claim 28, Nguyen teaches an A/V processing system that includes a video 
source (camera; Fig. 2, 200). Nguyen discloses a computer program product for recognizing 
gestures contained in video data, comprising instruction operable to cause a programmable 
processor, in communication with the video source to segment the video clip and automatically 
determine if the video clip contains a predefined gesture. See col. 1, lines 17-23. 

Nguyen determines a gesture window within which a gesture should occur. See col. 2, 
lines 23-25, which states, "The frame captures the person in the action of performing the gesture 
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at one moment in time" and col. 2, lines 28-34, which states, "These sequence of frame data sets 
taken over a period of time. . ." The method determines information related to a gesture 
occurring in the video clip only at the specified timing. See col. 3, lines 1 1-14, which describes 
"determining particular coordinates of the subject at a particular time ", (emphasis added). 

Nguyen does not explicitly teach an audio source to provide audio data including beat 
data from which beat data is extracted, such that the a specified time window of the segments are 
based on the beat data or obtaining the video data during a time that the audio signal is being 
produced. However, Suzuki teaches a dance game, which is intended to make the player use his 
entire body to create rhythm sensations (col. 1, lines 47-50). The dance game of Suzuki teaches 
that audio data is received and the beat data is extracted to create a specified timing window (col. 
7, lines 24-34). Suzuki determines if a predefined gesture (dance move) is performed by 
detecting dance steps on a step-on base only within a specified timing window related to the beat 
data. For example, see col. 14, lines 6-18. It would have been obvious to one of ordinary skill in 
the art at the time of the invention to modify the gesture recognition method of Nguyen to be 
used in the dance game of Suzuki, such that the video data is obtained during a time that the 
audio data is played and the specified timing window for seeing if a gesture occurred is related to 
beat data extracted from audio data as supported by Suzuki. One would have been so motivated 
in order to compare all movements of a player (i.e. arms) to an appropriate dance move because 
the step-on base of Suzuki only captures the movements of the player's feet, and does not 
recognize the movements of the rest of the body as intended in Suzuki. 
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Referring to claim 29, Nguyen teaches the computer program product includes instruction 
operable to cause the programmable processor to perform a Hidden Markov Model process to 
determine if the video clip contains the predefined gesture. See col. 5, lines 13-18. 

Referring to claim 30, Nguyen discloses a display (monitor; Fig. 2, 208) to display 
information based on whether the video clip contains the predefined gesture. See col. 6, lines 
36-44, which describes how figures on the display can be augmented if the gesture is recognized 
as a predefined gesture. 

Response to Arguments 
4. Applicant's arguments filed 10/12/04 have been fully considered but they are not 
persuasive. 

Applicant argues that Suzuki does not teach extracting beat data from received audio 
data. However, in order for the songs/audio data received in Suzuki to be accurately correlated 
with beat data as described in col. 7, lines 24-34, the beat data must be extracted in Suzuki. 

Applicant argues that Nguyen does not teach segmenting video data into a clip with a 
timing window corresponding to beat data. However, Nguyen does teach segmenting video (i.e. 
col. 2, lines 23-35, col. 3, lines 11-14, and col. 1 1, lines 23-26), and Suzuki teaches checking for 
a gesture within a specified timing of a beat, which segments the monitoring of the user/dancer. 
Therefore, it would have been obvious to one of ordinary skill in the art to segment the 
monitoring of user activity in the video gesture recognition of Nguyen in the same way, such that 
the system checks for a dance move within a specified time of the beat as supported in Suzuki. 
See col. 7, lines 35-50 and col. 1, lines 9-13 of Suzuki. 
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5. In response to applicant's argument that there is no suggestion to combine the references, 
the examiner recognizes that obviousness can only be established by combining or modifying the 
teachings of the prior art to produce the claimed invention where there is some teaching, 
suggestion, or motivation to do so found either in the references themselves or in the knowledge 
generally available to one of ordinary skill in the art. See In re Fine, 837 F.2d 1071, 5 
USPQ2d 1596 (Fed. Cir. 1988) and In re Jones, 958 F.2d 347, 21 USPQ2d 1941 (Fed. Cir. 
1992). In this case, the step-on base of Suzuki provides for a type of input means that utilizes 
full body motion for capturing a user's dance move/gesture. Nguyen teaches a type of input 
means that also recognizes full body motion and was known at the time of the invention to be 
substitutable for mechanical input means, such as the input means of Suzuki (i.e. col. 6, lines 34- 
36 of Nguyen). 

Conclusion 

6. The prior art made of record on form PTO-892 and not relied upon is considered 
pertinent to applicant's disclosure. Applicant is required under 37 C.F.R. §1.11 1(c) to consider 
these references fully when responding to this action. The documents cited therein teach 
tracking body movement. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Shawn M. Becker whose telephone number is (571) 272-4046. 
The examiner can normally be reached on M-F 8:30-6:00. 
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If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, John W. Cabeca can be reached on (571) 272-4048. The fax phone number for the 
organization where this application or proceeding is assigned is 703-872-9306. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 
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